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SECTION 40NE 
INTRODUCTION 



Disseminators within the National^Institute of Education's (NIE) 
Regional Program have become -involved in a widfc variety of activities 
aimed at improving education through facilitation of the widespread use 
of effective practices and products. Those who have designed and . 
implemented myriad dissemination efforts, both in the Research and 
Development Exchange (RDx) and within State Dissemination Capacity- 
Building Projects, have now begun to attempt to assess the impact of 
their efforts. This guide was produced by Northwest Regional Educational 

y 

w 

Laboratory 1 s Dissemination Support Service (DSS) to aid disseminators in 
that task. 

Purpose of the Guide ' i 

.The guide's primary purpose is, to assist disseminators in their 
efforts to evaluate impadt by providing suggested approaches within a 
framewQrk. Although a number of major studies regarding' dissemination 
evaluation have been published, reference guides for planning and ' 
conducting evaluation are not readily available. 

One important point should be noted, at 'the outset; the guide is not 
intended to help in writing objectives, but to work with objectives 
already defined. It is an objective-based effort, based on the 
assumption that existing written objectives are valid. 
Overview of the Guide 

There are five major sections in the guide: 

Section One provides an introduction and overview. 

Section Two reviews the general definitions in use regarding 
dissemination. 



Section Tfrree provides* a framework for determining impact 
and includes evaluative questions to be addressed throughout the process, 



> 



Section* Four provides an example of the application of the 

> 

framework using *a fictional situation. 
Section Five' provides a summary. 

The. Appendix contains an annotated list of resources. 
Guide Parameters 

Impact evaluation is an emerging interest. Because of this, several 

major issues w*illv remain unresolved. First, no clear, universally * 

** • *■ 
accepted definition of impact exists at this time in regard to * 

> 

dissemination activities. For the purposes of this guide, impact can be 
interpreted to mean the result of dissemination activities. 

4 

Dissemination itself is viewed as being a process by which R&D 
information-r-matar ials, projects, methods, and* so on— is moyed from where 
it originates to the potential user. The result of any disseminator's 

efforts can be categorized as effecting positive change, negative change, 

*>. 

or no'chenge'. The important point .is that the evaluation design should 

accommodate a process by which a determination can be made as to/whether 

any impact occurred and its positive or negative extent. Other complex 

considerations alsoerffcer into the impact definition. * Many other factors 

may attenuate the treatment; and methodology, because of the emerging 

state of the art, may not be well defined. A major complicating factor, 

as always, is dost limitations. Another factor is the 'time elapsed from" 

■ «i 
treatment to effect to later evaluation. The organizational level at 

which impact ahould be measured (primrary, tertiary, and so on), and 

whether any assessment of ^impact must take into account attitudinal as 

well as behavioral changes. are additional factprs to be considered. 

Decisions regarding management of these factors must be left up \o - 

\ ' * 



-individuals in determining their evaluation strategies. Despite these 

complications, however, it is envisioned that the guide will provide 

recipients with useful ideas for determining the impact of their 
dissemination activities. 
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SECTION TWO 

v < OVERVIEW. OF DI£SEMINATION 

( . . k y 

♦ 

4 1 

Ther6 has been considerable discussion and debate regarding what f. 
constitutes, dissemination. F6r purposes of this guide and its treatment 
of, impact evaluation, the range of activities described by the 
Dissemination Analysis Group (DAG) was accepted as the general definition 
. of what consitutes dissemination. Other descriptions and definitions 
were also reviewed in order to maintain a broa<3 view. The important 
* "point is' that disseminators strive to assist in the improvement of 
educational efforts; the activities in which they engage are the 
legitimate processes to assess in impact evaluation. 

The classifications system advanced by DAG includes: 
Level 1; Spread . The one-way distribution of knowledge in all its 
forms, including information, products, ideas and materials. Examples 
are publications and general mailings. * ^ 

Level 2: Exchange . The two- or multi-way flow of information, 
products, ideas and" mater ials. Examples are conferences and site visits. 

Level 3: Choice . The facilitation of consideration and selection 
among .those ideas, materials, practices* and other knowledge that can be 
used for educational improvement. Examples are traveling exhibits and 
"catalogs comparing alternatives. \ 
Level 4» Implementation . The facilitation of adoption and \ 

\ 

installation of improvements. Examples are consultation, technical \ 
assistance on site, and locally suited training programs. \ 

Another", more recent, delineation of dissemination techniques devised 
by Smith (1981) includes: 

1. Telling (as in articles, books, newsletters, presentations, etc.) 



2. Showing (using audiovisual mater iais 7 and 'by demonstration) 

3. Helping (through technical assistance and corfsultation) 

4. Training (through workshops, courses, etc.) 

5. Involving and intervening (working with the adopter) 

# As Smith pointed out, developers and researchers very often think of 
dissemination primarily as telling, usually through printed materials, 
and not often enough as one or all of the other activities listed above. 

One of the difficulties In determining impact or in designing 

r 

evaluation plans is -that some categor ies denote action by the 
disseminator, while others indicate action by the practitioner. The 
kinds of activities included range from those that move information from 
the R&D community to the practitioners to full-scale training efforts. 
This caveat is not intended to discourage evaluation planners; rather, it 

is intended to convey a sense of the complexity of the task and to 

suggest the need for careful and adequate planning. 
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SECTION THREE 

A FRAMEWORK FOR EVALUATING 
THE, IMPACT OP DISSEMINATION SERVICES 

The purpose of this section is to provide a structure or framework 
for evaluating the impact of dissemination. This structure is comprised 
of four dqj npone nts: f h . - 

1. Program Description » # , 

2. Evaluation Focus J 

3. Data Collection and Analysis 

4. Impact Assessment • 

The framework is presented in Table 1. Within each of the components 

{ 

are the steps it encompasses. Each of these, in turn, is broken down 
i,nt<J subquestiona to guide the evaluation procedure. JPhe actual content- 
of the questions, their major emphasis, and the way the answers are 
formulated 'are matters yhich can 1* fully addressed only with reference 
to a specific activity and the level of dissemination at which the 
activity takes place. ^' 
Program Description 

The identification ot these fpur elements constitutes * the program 

description: * 

• , What is being disseminated? 

• To whom .is it being disseminated? 

• For what purpose? * 

• By what means? ; 

r • ' " 



6 



Table I 

A. FRAMEWORK FOR EVALUATING THE IMPACT - 
/ OF DISSEMINATION SERVICES 



Program Description 

A. Identify what it is'thatt is being disseminated. , 

1. What 'services are provided? s 

2. What needs do th£y address? 

3. What influenced the decision to provide these services as opposed 
to others? 

B. Identify the recipient of the dissemination service. 

1. Who is the recipient? ^ % 

2. Was the recipient involved'in defining the need for the service? 

3. What is the recipient's linkage to the dissemination process? 
4* What is the recipient's role and responsibility in the 

dissemination process? 

C. Identify the objective of the dissemination. 

1. What is the primary intent of the dissemination? 

2. What are the secondary intents of the dissemination? f 

3. What needs are to be met ,by the dissemination? 

4. At wbat organizational level is impact intended? 

D. Identify the design for the dissemination. / 

1. What is the structural flow of the dissemination? 

2. What are the authority/responsibility relationships? | 

3. What are the provisions, for internal and external communication? 

4. What is the overall design of the dissemination program?' 

i ' 
Evaluation Focus 

A. Identify the audience of the evaluation information. 

1. Who wants evaluation information? 

2. What do, they want to ^know? 

3. Why do they want this information? 

B. Delineate the questions to be addressed. 



1. Wha't specific questions are to be answered? 

2. What 'success criteria will be used in determining the answers? 

3. What will be the evaluation unit of analysis — a single se^ice o 
cluster of services? £ 



1 



C. Assess practical issues in £fce evaluation. 

* ' L. Are^ evaluation costs within the range of available funds? * 
2 What is the timeline for completing each step of the evaluation? 
3. Can information gained. in prior evaluations be coordinated with 
present needs? r 

*D. Determine information sources and instruments. . 



1. \What information is needed to respond to the <^iestions? 

2. Who can provide this information? 

3. What data collection method will be used? y 

4. Will an existing measurement instrument be used or a new one 
developed? * . 

5. Will sampling be used? - 

6. Wi^l program implement at ibn as well as program outcomes be* 
measured? . * 

Data* Collection and Analysis 

<A. Implement data collection plan. 

1. Are there unforeseen difficulties in data collection? 

2. Are unintended effects apparent during data collection that are 
not provided for by the evaluation? 

B. Analyze data. * 

1. Is the analysis of data consistent with questions addressed?* 

2. Would it be useful to perform a secondary analysis — that is, to 

9 ' analyze the data in a different way to address different types of 
questions? ^ 

' Impact Assessment 



A. D^ermine'the impact through jthe measurement instruments chosen. 

1. W^re specific objectives, achieved? 

2. Here there any positive unintended effects? 

3. Were the^e any. negative unintended effects? 

4. Was th<Kprogr^am implemented as intended? 

B. Draw Conclusions v 



1. How close did thg^actual impact qome to the intended impact? 
k .2. wriich factors facilitated achievement *of objectives? Hinder 

3. What was th^role of unintended effects? 

4. What was the overall success of the service? x 

5. ' Which segments should be repeated? Discontinued? Refined? 



ed? 



Whether one is evaluating a system" of. dissemination or/ a component of 
a system, it is important to provide a complete description of what it is 



that i* being evaluated. The impact of/ the dissemination will be judged, 
a: least in part, in light of irtformatiGn provided in the program ^ 
description about goalfc and ^objectives, design^ and processes. 

A worksheet is provided (Worksheet^ 1) to help organize the program 
description. If several services are being provided and the evaluation 
cannot cover Ihera all, enter a priority ranking to determine which are 
the most important evaluation targets. \ 

-Evaluation Focus * - 

Fpcusing the evaluation entails identifying the audience and purpose^ 

'for the evaluation, and specifying the areas /of the'program that will be 
the focus of the evaluation. It also includes the Consideration of 
practical issdis such as cost and time, and the planning of data 
collection and analysis; Work^eet 2 covers many of the major factors in 
a good evaluation plan. This sheet should be, filled -out ' as the plart is 
developed. 

Audience . When an evaluation is undertaken, the planners nee^ to 
keep- in mind from the start who needs the information and what their 
needs are. In this way, the chances of eventual utilization of the 
information can be maximized.* Often there are two or ; morfe audiences, . one 
of which is a primary audience. For example, a primary audience .might be 
the funding agency, while the project director or other host organization 
administrator^ might consitute secondary audiences. These different 
audiences could have very different information neecjs. It would be to 
the advantage of th\ program director or the evaluation team to /determine 



Dissemination Service 
4 1 * 



Worksheet 1 / 
* Program Description 

-pEFINING THE DISSEMINATION PROGRAM TO BE EVALUATED 



Recipient 



DAG Function 



Objective 



Priority Ranking 
for Evaluation 



- \ 



T 




Worksheet 2 
Evaluation Focus 

EVALUATION PLANNING SHEET 

Audience for ' — Collection Data Analysis 
the Evaluation Evaluation Question Data Source Method Instrument Plan_ 



V - 

2 -, <l 



*1 




4. 



5. 



t 9 t. 
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these information needs very early in the development of the program in 

order to ensure the collection of the necessary data duri^tg the 

appropriate stage of program development. 

Evaluation questions . Frequently, limited resources— including 

time— do not allow for an evaluation plan which addresses all questions 

which might be of interest to the various audiences. Thus, the manage^ 

or evajuator must negotiate with the relevant audiences the major 

questions to be addressed. In addition, the evaluation team must meet, 

with the relevant audiences to determine the range of possible answers to 

the evaluation questions to be considered; to formulate criteria to be 

4 m 
employed in judging success or failure? to clarify the assumptions tp be 

» 

made (i.e., those pertaining to design, analysis and measurement); to 

define, the system in which the evaluation is to occur jjand to determine 

the policy and contractual agreements which wiSS: govern the evaluation 

work. These things should.be accomplished during the planning of the 

..study, but should also be\6pen for reconsideration thro^hout the study. 

- in evaluation planning a decision is often required regarding what 

the unit of analysis will be. In dissemination terras, this question 

refers to whether the evaluation will focus on single-service events, 

'combinations of like events, a series of relied events, the entire State 
m 

Capacity Building Project or Regional Exchange, and so forth. Often 
several services to clients, such as mailings, workshops and . 
individualized conferences, will be'related to the sam^ overall objective 
agd, in such cases, 'it is often more relevant to focus on them together, 
1 as a unit. Cost considerations can also favor evaluating a cluster of 
services rather than a series of individual ones. If services are to be 
oo«bined for evaluation purposes, then questionnaires interviews and 
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other data instruments can inquire about the effectiveness of the program 
rather than each of its components, "and results can be 'sought following a 
series of services rather than each' one individually. 

Two worksheets (Worksheets 3a and 3b) are provided to aid in the 
determination of the unit of analysis. If one is uncertain what the unit 
will be, Worksheet 3a should be used first. In the left hand column list, 
all the dissemination services that are to be evaluated. For each of 
these', in the middle column list the service's objective. Those services 
with similar objectives are likely candidates for grouping into a single 
unit. In the third column, indicate your grouping scheme.* Worksheet 3b 



is a summarizing 



izing ^ool. 



At the' top of the page list the conceptual unit 



and its objective. Underneath this; list all the component services that 
are included in the unit. t 



/ 
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Worksheet 3a t 
DETERMINING THE EVALUATION UNIT OF ANALYSIS 



List of Individual . Objecti^ of Grouping . 

Dissemination Services the Service « - Scheme* 

i. 



A. 



4. 



5. 



• r . 



*DOTE: Grouping may occur around clusters of services pertaining to a single 
objective or around similar objectives. ' 
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* Worksheet 3b 

SPECIFYING THE UNIT OF ANALYSIS 

Unit of Analysis (description of servipe) : 

* • * 



Objective: 



Is the service to be evaluated: 



A single -case of a provision of a service? - 

A cluster of similar events? 

General services for^all clients? . ^ 

The whole program? , 

% Other (state): . 

Component service description : 
^ 

Service (list) Objective Recipient 



T" 



1. 



r 



3. 



i 



A. 4 



5. 



< 
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Practical concerns: Cost, timing and previous efforts . Evaluation 
costs should be determined at the planning stage so that the study's scope 
can be assured of fitting the ava liable resources. Be certain to include 
all costs related to evaluation activities, including salaried, benefits, 
consultants, equipment, office supplies, use of computer time, travel, 
telephone and postage, reproduction costs, and so on (Kiefert and Maser r 
1977). A checklist (Worksheet 4) ~has been provided to help organize these 
cost considerations, * The available evaluation funds, of course, should 
exceed the expense estimate — with a wide enough margin to allow for 
unexpected costs. If funds are a problem, the budget may have to be 
tr lamed or a sub-area of the planned evaluation may have to be dropped. 

The schedule for the evaluation should be planned to. whatever degree 
possible at the outset." Generally, the final completion date is 
specified, and various stages along the way (such as data collection) must 

be planned and carried out according to careful scheduling. The other 

i 

steps in the evaluation should be planned to coincide with these 
requirements. A timelog has been supplied (Worksheet 5) which can be 
useful in scheduling. For each of the major evaluation steps, the 
starting date and fcompletion date should be filled in* 



\ 
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Worksheet 4 
ASSESSING THE COST OF THE 'EVALUATION 

Funds available for the evaluation: 

Estimate of expenses involved in the evaluation: 

Staff salaries 

Consultants 

Data Instruments 

Telephone and Postage , 

Equipment 

Computer Costs 

Travel m 

Other (specify) : 

9 



TOTAL: 

Do available fun^s exceed the planned 
expenses? 



1 
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Worksheet 5 
EVALUATION TIMELOG 



J 



Activity 



Starting 
• Date 



Completion 
* Date 



1. Describe the dissemination program to be 
evaluated. J 

2. .Clarify the goals and purposes oi the evaluation. 

a. Identify audience 

b. Delineate evaluation questions 

c. Assess practical issues 
d. 

3. Implement the evaluation plan, 
a. Assemble data: 

Data source A (describe) 



etermine data sources 



Data source B (describe) 
Data source C (describe) 
b. "Analyze data,. 

s 

Assess program impact and report the results, 

a. ~ Determine impact and draw conclusions 

b. Report orally (if appropriate) 

c. Final report 



■erlc 
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If your organization has previously undertaken scofe evaluation 

efforts, such as surveys, interviews or case studies, these can prove 

i 

useful in the current evaluation plan. First^ the development work might 
be generalizablej th* instruments previously used might be appropriate 
with little or no modification, or perhaps effort could be saved by 
referring to pre'vious analysis designs, computer programs, and stf^on. 
Secdnd, the information the previous studies supplied might in seme cases 
be \interpretable as baseline data. On the other hand, in many cases, past 
work will not be pertinent to present needs and will contribute little, 
making it necessary to proceed independently of the prior efforts. The 
utility and generalizability of prior evaluations must be decided on a 
ca?e-by-case basis. ' » n 

Information sources and instruments . Once the practical issues have 
been assessed, one's attention should- turn to a plan for collecting the 
necessary information. The Jcind of data one feeds can be determined from 
an examination of the evaluation qu^tions. Examples of different kinds 
of data that might be useful are requests for mailings or consultations, 
workshop evaluations and fQllow-up studies on information utilization^ 

district staff attitudes, success of new program implementation, etc. The v 

i 

source of the data can likewise be directly determined. For example, a 
question about the impact of a catalog of current materials disseminated 
to teachers would suggest teachers or someone working with teachers as the 
most likely source of information. Sometimes it will be useful for 
purposes of objectivity to collect information from different populations, 
such as teachers and administrators. 

Several data collection methods are often applicable to a particular 

4 

question, and one can draw from such techniques as interviews, 
questionnaires, existing records or, in some cases, observation. Each of 

' 19 
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these carries its own unique advantages and disadvantages , and the reader 
is referred to an evaluation handbook (e.g., Faddis, et al., 1981) for a 
fuller treatment. Again, drawing from Several technique^* (such as 
\ interviews and existing records) can give a more complete picture than 
any one method alone. When the choice of collection methods is made, one 
can determine whether a measurement instrument fcas^%o be developed for 

\i r r 

the purpose or whether an existing one is suitable. 

Another conSTtteration is deciding thja breadth of the analysis; will 
sampling suffice? A monthly newsletter published by Capitol 
Publications, Inc., entitled How To: Evaluate Education Programs 

; — 7 

(July 1980), suggests that correct use of sampling procedures may save 
time and money and also avoid inconvenience for a number of people and 
lists several points to consider. Sampling is suggested when: 

• The cost and difficulty of getting data from all clients is 
prohibitive. 

• Mathematical precision is not 'essential for drawing conclusions. 

• The information yielded by sampling >will be adequate for the 
purpose and audience for the evaluation. % 

• It is posssible to use proper sampling methods. 1 
In addition, the evaluation should include a^means for measuring the 

degree to which the program was implemented "as intended. In this w^y, in 

- r 

the case of negative resulrts one will know whether the program was poorly 

i 

planned or just poorly implemented. 

Data analysis plan . A final step in planning is deciding on the 
analysis plan. One might require a statistical test between groups of 
scores or simply a tabulation of respondents and percentages. The choice 
o*f information type and measurement instrument will dfet^rmine the 
analysis to some degree. 
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Worksheet 6 has been included as a review sheets f or evaluation 
plan* A good plan should incorporate as many of the feasibility factor 
^in Parts A and B as possible. Part C relates to prior evaluation 
efforts. These efforts may or may not be useful for current concerns* 



4 



1 . 



<ERIC 



21 



■1 



Worksheet 6 
.EVALUATION PLAN REVIEW SHEET 

f 

€ Feasibility of task to be evaluated . 

'Is the task: Central to your work? 

Frequently performed? w 
One that affects many people? 



Open to alternative styles of management? 
Affordable^ evaluate? 



(Answers should be affirmative.) 



B. Feasibility of the evaluation plan . * 
Does ihe plan: Address current concerns? 

Accommodate existing resources and timelines? 
Avoid disrupting normal activity?.^ 
Promise to yield the data you need? 
-^Answers should be affirmative.) 

C. Continuity of evaluation efforts . 
Should past evaluation efforts continue? 

Are past efforts used in the «current t plan? 

How can yoy build on current effort for future evaluation? 

. How can evaluation be integrated into other activities? * 
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Data Collection and Analysis m 

< Data collection . The collection and analysis of data should proceed 

according to plans' developed in the preceding step. H*ever, during the 

actual data collection process, it may be necessary to modify the data 

collection plan. Unforeseen obstacles (e.g., missing data files, low 

response rate, unexpected time considerations, etc.) or unanticipated 

observations (e.g., discrepancy in what was expected to be going on and what 

was actually going on) may make it necessary to change the study sample, 

limit or expand the scope of data collection or modify the plan in some 

other way. In these cases the data collection plan should be adapted to the" 

r 

^situation, rather than vice versa. 

Data analysis . Data analysis involves (a) cleaning the data, (b) 

> 

analyzing it as planned, and (c) re-analyzing it should it become useful to 
do so. In cleaning the data, the concern is the elimination of misleading 
or biased jinformat ion. Por example, data provided by a principal in a 
questionnaire that* was meant for teachers should be eliminated if the 
objective is to report how teachers feel about the dissemination. Another 
exawple would be the discovery of inappropriate response patterns by the 
sample', perhaps through misunderstanding of the directions. Any irregu- 
larity that might cause" misinterpretation of results should be eliminated 
from the analysis . The data analysis technique should have been defined in 
planning the evaluation study. At this point it is a matter of conducting 
the analysis as planned. In addition, new relationships among the data may 
appear that suggest new forms of analyses. 'Assuming that time and funds are 
available, It is recommended that secondary analysis of the data be done to 
maximize the use of available data. Frequently, other kinds of questions 
which might not have been apparent or important during the planning of the 
study can be addressed by secondary analysis of existing data. 

23 



A data analysis checklist has been provided (Worksheet 7) . If the 
evaluation proceeds exactly as planned, each of the ^x questions should 
receive a NO response. However, it is likely that" there will be a change 
necessitated somewhere in the data collection and*1inalysis. If so, aome 
adjustment in the analysis wi^^ required— e.g. , a low initial response 
rate might lead to further mailings before any analysis begins, or 



nonuniform data ^collection might require some of the data to be 
invalidated. Por these design or analysis problems, it is best to 
consult with an evaluation expert on the staff or in the field. 
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Worksheet 7 
DATA ANALYSIS CHECKLIST 

Were any of the planned respondents unable to 
participate in data collection? 

If Y1S, describe: 

V \ 

Course of action: > 



Were data collection procedures nonuniform 
in the sample? 

If YES, describes 



Course of action: 



Here data instruments incorrectly filled out: 
y YEfi, describe* 



Course of action: 
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Did any difficulties arise in the data 
fcollection that might have compromised 
the validity of the data? ■ 

If YES, describe: 



Course of action: 

\ 

) 



Did any difficulties arise in data analysis? 
If YES, describe: 

r 

Course of action: t 



Was a secondary analysis called for 
when reviewing the results? 



If TBS, ^es 



cribe: 



Course of action: 

c 



I 
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Impact Assessment 

r- * 
The final stage of the- process is the assessment of impact. 

Worksheet 8 has been provided, to assist with the organization of this 

stage.. First, the answers to the specified evaluation questions are 

determined from the data analysis. When ti^ instrumentation, methodology 

and analysis have been designed to reflect the evaluation question, this 

is a fairly straightforward procedure. For instance! if the evaluation 

question concerned whether staff attitudes improved as a result of a 

series of workshops, a post-program attitude measure used in comparison 

with pre-progrim scores will provide relevant information. In this way 

it cm be judged whether specific objectives' were achieved. 

On the other hand, if there were 4ata collection problems 

necessitating a change in method, additional caution must.be exercised in 

y 

determining answers. For example, suppose that the sample of respondents 
had to be restricted after the start of data c»llecti^n^^ o^der to 
verify the new sample's relevance to the original evaluatiorNjuestion, 
the evaluation team must be sure that the adjusted sample is still 
representative of Jche population of interest— i.e. , did each member of 
the population have an equal chanoe of being selected into the sample? 

The roles of unintended effects, positive or negative, must also be 
considered. For instance, through interaction with program- staff during^ 
data collection, an evaluator might have found that staff attitudes were 
tremendously improved, even though' the data instruments were designed 
only to measure frequency of ihter action. This finding should be 
included in any observation report that comes out of the evaluation, and 
should be considered in drawing conclusions regarding the future of the 
program. J 
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Worksheet 8 
IMPACT ASSESSMENT * SUMMARY FORM* 
Evaluation question (state): 

1. Was a clear answer obtained? (state) : 

2. Unintended positive effects (if any): 



a. Bow valuable is this effect? 



> - 



$ 

b. Is the effect stable? 



3. Unintended negative effects (if any) 



a. How serious is this effect? 



b. Is the problea resolvable in "future applications? 



4. Overall nature of impact: 

5. Conclusions (outline): 



♦MOTE: Use one page for each evaluation question j 
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. The implementation of the dissemination plan should be checked along ^ 
with outcomes. This can be <\one through documentation of the program's 
activities. Too often interventions have been judged to have had little 
or no impact when in fact the problem was that they were never 
implemented as planned. With information available about ?the extent of 
implementation , it will be possible to make more informed decisions about 
whether it is necessary to change the dissemination plan or simply 
implement it more faithfully. ^ 

After discovering the actual impact pf the program, it is possible to 
draw conclusions relevant to 'subsequent decision making. This step is 
distinguished from the previous one in that it is more broadly focused. 
The disseminator must review the evaluation', explain the obtained 
results, and make decisions regarding future actions. Specific 
considerations in this process would include: causal linkages between 
prog*0fc elements and results; which program factors facilitated (and 

- P 

which hindered) achievement of the objectives; the presence and 
significance of unintended effects; the implications of any secondary 
analyses that were carried out; ancl, finally, which secants should be 
(continued, modified or dropped. * 
The procedural outline presented here is considered 4 to be a framework 
because it is meant to guide rather than restrict the process of impact 
evaluation. Individual programs will present particular considerations 

4 

that may change the emphases offered here. The next section provides an 
example of how impact evaluation might be conducted. 
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SECTION POOR 

K 

* . AN APPLICATION OP TflE TOAMEWORK 

In order to illustrate the use of the framework, a simulated example 

is presented. This example is meant to illustrate the steps discussed in 

Section Three. The reader may find it useful to refer to TabJ.e I when 

reading thia Section. It will also be useful to refer to the completed 

wotksheets which have been provided for illustration. It must be 

r 1 t 

remembered, of course, that each situation will present specific details 
that will change the application of the framework. 

One factor, for instance, that will influence the form and planning . 
of the evaluation is the source of the perceived need for the study. If 
internal project personnel have raised the questions of impact and are 
seeking answers for internal formative purposes, the planning phase of . 
the study is simplif iedj^decisions regarding the statWg of objectives 
and measurement instruments do not 'require outside consultations. This 

• is not meant to discourage the seeking of input from various sources, nor 
does it mean that less clarity is needed in the planning phase. But the 
steps of matching the evaluation questions, measurement and analysis 
procedures to the persons making eventual decisions are crucial. If the 

' impact questions have been raised by other agencies or will be of 
interest to them— e.g., funding agencies' or host agencies— these 
organizations must be brought in at the start to obtain the maximum 
possible assurance that the study will be meaningful to them. 

The example presented here involved a State Capacity Building Project 
entering its final year of federal funding. Project personnel felt 
J confident in their relations with local school districts and believed 
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that their information dissemination activities were useful to some 
extent. Nevertheless, the extent to which LEAs actually used the 
information they received in local policy planning and decision making at_ 
the district or school level had not been studied systematically. The 
original qontract called only for an evaluation of the implementation of 
.activities. As projected, the State Capacity Building Project had * 
established a Materials Center and links to the ERIC system, and a 
procedure fox responding to LEA requests. Now "the agency's personnel 
were interested in obtaining d^^on the kind and extent of positive 
impact occurring as a result o^heir program's dissemination 
activities. They hoped' that the State Education Agency would continue 
the project after federal funding expired, and they felt that strong data 
supporting local impact would be very helpful. 

* * The project personnel first assembled an evaluation team. They 
contracted with an outside evaluator and assigned one of their own staff, 
to work on the evaluation as well. They requested consultation with 
their Regional Exchapge office, and the Rx assigned an tnSividual to work 
on the evaluation part time. Finally, they asked for input from the ^ 
director of the Professional -Services Division of the SEA, since they , 
realized that information resulting from the evaluation would eventually 
be considered by the SEA in any continuation decision. At that later 
point, it would have to make sense to state agency administrators, and so 
the director's input was requested early in the process. He was able to 
attend most of the planning sessions and kept in close contact with the 
evaluation team as the study progressed. 



In beginning the evaluation, the evaluation team first turned its 
attention to the program' description. The service to be studied was 
identified as the project's normal procedure for handling information 
requests. Ideally the project would determine the impact for all of its 
dissemination activities; but because of limitations of time, money and 
personnel, the evaluation team had to limit its scope. Responding 1 to 
information requests was one of the major activities of the agency. 
Normally, when a district or school sent in an inquiry, the project 
conducted a computer search of the ERIC filed and its own Materials 
Center and sent back an output of article abstracts resulting from the 
search. The clients would look through these abstracts and send back a 
letter identifying which articles or documents they felt were 
particularly' relevant to their needs. The project people would then send 
the articles requested — which would complete the interaction unless 
further requests were made* y » 

The impact of this project activity was considered relevant to both 
potential audiences. Prom the SEA's point of view, a demonstrated impact 
would be a strong argument in favor of continuing the project when its 
federal funding expired. Prom the State Capacity Building Project's 
point of view f the results of the study could have bearing on whether the 
current procedure should be continued, 'refined or scrapped in favor of an 
alternative one. 
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Dissemination Service 



v Worksheet 1 

Program Description 



DEFINING THE DI S SEMINAT IQN PROGRAM TO BE EVALUATED 



Recipient 



DAG Function 



Objective 



i. J^^fyf^fy^ 




2. 




Priority Ranking 
for Evaluation 



-%0j - — — ^^^^ 



Ul 
Ul 



i t 
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The recipients of services were administrative personnel ih the 
school^ and districts of the project area. In tlH^case the 
identification of recipients was not difficult since the service is a 
reactive ode and contact is initi^tdCd by the client; this decision can be 
more complex in other cases, it is a good idea to view recipients as 
, specifically as possible-- i.e., individuals or offices. If the clients' 
identity is' vague (e.g., Tall schools in^^state"), your decisions on 
where and how to adminis*»er the evaluation might be more arbitrary. One 
question that did arise* was whether the "recipients' 1 should include - 
potential requester^ as well as actual requesters. The evaluators 
decided that since this service is a reactive, responsive type, only 
those districts making requests would be considered in the amtolationr of 
interest.^ The quesAon was not one of proactive services-^^Sas whether 
those people requesting information actually made use of it later. / The 
proactive dissemination activities, although equally important, would 
t&\l{tofoz another category * ^ * 

' Betermining the goals and objectives ' of the service required some 
careful focusing. At first glance oh^p»ight think that "SifccAsfful 
impact" would translate ir)ta a change in all or most cases. Van. 

intriguing topic olten turns out <>n inspection to be not what one needs 

> • 
, or wants, a phenomenon not limited to evaluation practice. The 

conscientious local administrator can and should keep up on new 

educational developments, not all of which will be relevant for 

utilization- in that disttict— the state capacity project aids ttfht $ 

awareness process. 1 Therefore the evaluators decided that "impact" of 

their service should be viewed in terms of whether the service increased 



( 

6 » 

»' ( 

% 

t 

knowledge, reduced uncertainty, and aided in local decision making. In 
this case* the evaluators stated the goals this way: 

GOAL; provide a linkage service enabling local educators to 

increase their knowledge of new devel opine nts in all areas 
of educational thought and practice. 

V . / ' 
OBJECTIVES : 

• To pVovide local educational personnel with the means of 
obtaining a moderate or wide knowledge of any topic 
relevant to their administrative or informational needs. * 

To provide an efficient inf ormation*service that can ease, 
inform ^nd improve the quality of local decision making. 

The next series of steps involved focusing the evaluation. First, in 
light of the funding considerations previously described, the audience 
for this information was identified as being primarily the SEA. The 
information would also be gathered so as to be useful to people within 
the project itself, so project personnel were considered a secondary 
audience. * * 

Ihe next step was to establish the evaluation questions . Given the 

previous considerations, the evaluators decided to concentrate on two 

factors. The first factor Was 'the clients' satisfaction with the extent 

of the information, provided; that is, could they obtain -rnfoTmat ion about 

their topic to a desired depth and specificity? The second factor was 

the extent of utilization of the information in decision making. The 

questions they decided on we're these : . 

1. To what extent do clients perceive their informational needs 
satisfied by the information distribution service? 

f To "what extent is the information provided by the service 
incorporated into the decision making process at ttte local level? 

Through discussion the team agreed that 75 percent of respondents 

expressing satisfaction in question 1 and 50 percent of respondents 

reporting information use (question 2) would be Considered successful for 



the proje^. 



4,' 
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Worksheet 2 
Evaluation Focus 

EVALUATION PLANNING SHEET 



Audience for 
the Evaluation 



Evaluation Question Data Sourpe 



Data Collection 
Method 



Data 
Instrument 





2. 



3. ' * . # ol*U44*r?. 



4. 




Analysis 
Plan 



J 

* f 

Once the program description 'was clear, the evaluation audience was 
identifie^, and the evaluation questions were specified, the evaluation 
team could tgrn ijts attention to the measurement procedures, i.e. , the 
information sources and instrument-* . The evaluators decided to follow up 
on a sample of the requests that arrived during the fall of that school 
year. The office generally received about 10 to 15 requests for 
information a month. The ^motivations for the requests varied from 
specific needs plating to a teacher or school's planned instructional 
changes to an educator's general desire to keep inf ormedf^^he requesting 
individuals consisted of teachers, principals, curriculum directors and 
linkage facilitators located in the district. The team decided to draw 
dp a questionnaire to be sent to individuals making the request, asking 
- how often they had used the service before, what their aim was in 
requesting the information, and hpw closely the information need was tied* 
to specific policy or instructional matters. This questionnaire would be 
sent to a wide sample of the clients, representative of all the requests 
in terms of regions and personnel making the contact. The evaluation 
team then would plan follow-up investigations for some of these cases, 
particularly th^e^trr-which the respondents indicated a definite need. 
These follow-ups would consist of interviews with key local people and 
review of existing records, such as lesson plans, that might reflect the 
utilization of the new information. 

As noted, the planned analysis was straightforward, consisting of 
tabulated percentages and content analysis of the interviews. The 
evaluation team considered what outcomes were possible and how ready they 
would be to accept impact information, both positive and negative. For 
instance, if the investigation indicated that use was not being made of ^ 
the information, the team recognized that this feedback would be accepted 

37 



L 



as a realistic picture, however unfortunate the conclusion might be. 
Credibility is severely compromised by an evaluation process in which, 

following positive feedback, the program personnel claim Success but 

i 

following negative feedback, they criticize the study or start' a new one. 

The framework in Section Three also includes consideration of the 

degree of implementation of the dissemination service. The point here is 

that an evaluation team wants to make sure that whatever conclusions it 

eventually derives about program effectiveness are based on the program 

its^f, operating as planned. What often happens instead is that the 

program never pally gets carried out as intended, ftr any of a variety 

of reasons including funding changes, poor management, over- or 

underresponse by clients, knd so forth. When this happens, a^negative 

evaluation is usually th* result, *and it' is important, for -an evaluator to 

know that the failure is not due to the originally planned program. This 

is particularly important for formative evaluation efforts in which the 

evaluation gets under* way at the same time the program does— that is, at 

a point when the evaluation* team may not have a clear idea of what the 

program will look like in operation* 

In our case study, the information distribution service*had been 

« 

operating for two years, and so the evaluators knew there was little 
chance for any surprises to occur which would interrupt its normal 
fi^ctioning. Nevertheless, for the fall term which the evaluation was to 
.cover, they carefully monitored program operation. They mad^sure that 
the request frequency was similar to years past, that the computer* was 
workipg_and, in general, that the service was progressing as expected. 
This gave the« added confidence in the validity of their later follow-up 
■easure. 
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Worksheet 3b 
SPECIFYING THE UNIT OP AMALYSIS 



On it of Analysis (description of service): 




Objective: 



\ is the svvice to be evaluated: 

A singfe case of a provision of a service? 
A cluster of similar events? 
. General services for ali clients? - 
The whole program? 

Other (state): 



Coaponent service description ; 
Service < 1 ^« t ) 



* Objective 



Recipient 



2. 



3. 



5. 
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Worksheet 4 
ASSESSING THE COST OF fHE EVALUATION 

Funds available for the evaluation: 

, « 

Estimate of expenses involved in the evaluation: 

Staff salaries frt~j) 

Consultants 



Data Instruments 
Telephone and Postage 
Equipment 
Computer Costs 
Travel 

Other (specify) : 



' /H) 



3o 



0 



bo 



TOTAL: 



Do available f \mdtf' exceedf the planned 
expenses? 



• c 

Worksheet 5 
EVALUATION TIMELOG 

Starting Completion 



Activity 



Date Date 



1. Describe the dissemination program to be 
evaluated. 




2. Clarify the goals and purposes of the evaluation, 
a. Identify audience _ 
, b. Delineate evaluation questions _ 



n 



c. Assess practical issues 



d. Determine data sources 
Implement the evaluation plan, 
a. Assemble data: 



II 



Data iourc< A (describe) - . J^Lrff j fa^/tfl 



Data sources (describe) 



Data source C (describe) 



b. Analyze data. tj tl 7?U^ iff I 

s 



4. Assess program impact and report the results 
a.' Determine impact and draw conclusions 



b. Report orally (if appropriate) 

c. ,1-1 report . \ J^t if SI Qk*fy Mt 
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Worksheet 6 
EVALUATION PLAN REVIEW SHEET 



A. Feasibility of task to be evaluated . 

Ifc the task: Central >to your work? ^ 





Frequently performed? 
* One that, affects pany people? 
Opea to alternative styles of management? TV ^/x^Lc(^ 
Affordable to evaluate? 
(Answers should be affirmative.) 

B. Feasibility of the evaluation plan . 

w ' /• 

Does the plan: Address current concerns? V 

Accommodate existing resources and timelines? 
Avoid disrupting normal activity? 
Promise to yield the data you need? [/^ 
(Answers should be affirmative.) 

C. Continuity of evaluation efforts . 

Are past efforts used in the current plan? 



How can you bulla on current errort ror iuiucb evaiuauuui 7/ . 

Hoi/can evaluation be integrated into. other activities? 
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The third part of" the framework concerns data collecting 
ind analysis . Aftej: the fall term ended, the evaluation team found that 
60 information requfe^ts had been made. They agreed on a sample of 35 for 
initial follow-up by questionnaire . These represented the full 60 in 
that all of the districts were included? there was diversity in the 

itions of local personnel initiating the requests; there were new and 
experienced users of the service; and thepe were both further inquiries 
and no inquiries' for full articles following the mailing of tfie ERIC 
search. Through the sampling processes ^involved in the data collection, 
it was again very helpful to have the participation of outside people to 
dispel doubts that the sampling was not biased toward positive results. 

After receiving the 35 completed questionnaires, the evaluation team 
tabulated and categorized the responses. They chose seven for follow-up 
site visits. Two of these irrficated strong use made of the information, 
two indicated little use, and three reported moderate degree of use. Let 
us examine tow the follow up was .conducted for one of the two high-use 
cases. 

At the site studied, the individual who had made the inquiry was the 
curriculum director of a schi^Tshe had asked for ar ticleiTnealing with 
the relationship between math achievement and time-on-task, and a 
variable called "academic engaged time." In the questionnaire she wrote 
that she had been only moderately satisfied with the completeness of the 
articles from the search, and she supplemented these with inquiries made 
at the education department of the nearby university. She also said tha£ 
the information she gathered had been very useful at her school. 

One of the evaluators contacted the curriculum director and scheduled 
a morning visit to the school for interviews that tould be inducted with 
her, with the .pr incipal , and with one of the math teachers. In each of 



these discussions the interviewer asked what the initial motivations were 
♦for gathering the information, how the information influenced decisions 
that were made .inside .the school, how school conditions changed as a 
result of implementing the decisions, and how the process could have been 
improved. In addition the intervieVer requested materials 'and records 
that would reflect the nature and extent of the change. 

In the course of the discussion, the evaluator learned that the 
school- personnel had been concerned about lower-than-average math scores 
on standardized tests, and decided to seek information on how they could 
alleviate the problem. The curriculum director had read a research 
report that showed there were surprisingly large differences in time 
allocations for math instruction^ among districts, schools $nd 
classrooas. In addition, there was a difference in the nature of 
instruction such that "academic engaged time" varied even when class time 
x allocations were comparable.. The director and principal had two meetings 
with teachers in which this information was presented and discussed. As 
a result of this process, two actions were decided upon. First, the 
director cohtractec} with a knowledgeable university educator for a 
workshop instructing teachers about ways to increase the quality of 
•academic engaged time" during math instruction, 'second, the principal" 
made a change" in the average allocation for math instruction per day from 

* 

40 minutes to 60 minutes. The capacity project's information had been 

i 

used in the following ways: 

1. It increased the knowledge of school personnel regarding the 
potential influence of the variable in question. 

2. It helped them determine that their daily time allocation to math 
was slightly below average. 

** 

3. It helped the curriculum director in her search for an educator 
knowledgeable about that line of research. 

c 
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In their interviews the curriculum director, the principal and the 
math teacher all expressed optimism about the project, but it was still 
too early to determine 1 the results of the change. The evaluator was able 
to obtain from the school several records of .the change: the agenda and 
other materials from the workshop that was conducted, and a sample of 
several teachers' daily math lesson plans, before &nd after the period in 

w 

question • 

After all the interviews at the selected sites .were conducted, the 
evaluation team organized all of its questionnaires and interview data to 
reflect the sample as a whole. With regard to the first evaluation 
question, they found that of all the respondents, about 30 percent were 
very satisfied with the extent of the information they received, about 40 
percent were somewhat or moderately satisfied, and about 30 percent felt 
#fc there was substantial room for improvement. Regarding the second 

evaluation question, they found that about 20 percent of their sample 
reported no subsequent utilization of the information, 50 .percent 
reported a beneficial gain in awareness among staff, but not of a level 
leading to a decisive role in policy making or practices, and 30 percent 
reported that the role of the information in changing relevant practices 
was very important. The evaluation team also found that the chances of 
utilization were increased when an administrator made the request/ when 

the information was shared with several people, and when the inquiry was 

\ 

based on a specific, identifiable need, 1 
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"Worksheet 7 
DATA ANALYSIS CHECKLIST 



Were any of the planned respondents unable to 
participate in data collection? 

If YES, describe: 



YES NO 




a 9 

Course of action: 



Were data collection procedures nonuniform ]/ 
in the saaple? _L 



If YES, describe: +~ * s/ s-/ 

Course of action: , ' _ ^ '+ ' 



Were data instruments incorrectly filled ogti 
If YES/ describe: * 



Course of action: 



Page 1 of 2 
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YES NO 




4."' Did' any difficulties a'rise in the data 
» * collection that might have compromised 
thai validity of the data? 

<~ 

If "YES, describe: 



Course of action: f 

5. Did any difficulties arise in data analysis? * 
If YES, describe: 



«5. 



<5? 



6. Was a secondary, analysis called for 
* when reviewing the results? 

If YES # describe: 



.Course of action: 



4 



v 



V 
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Worksheet 8 

IMPACT ASSESSMENT: SUMMARY FORM* * 
Evaluation question (state): jt^Ui^ x^/^^^ -rU£^o-~ 

1. Was a cleat^Ei#er obtained? (state) : 

2. Unintended positive effects (if any): A 



TT^pow valuable is this effect? 
• b. Is the effect stable?' 

3. Unintended negative effects (if any): 

a. How serious is this effect? 




i 

b. Is the problem resolvable in future applications? 




5. Conclusions (outline) v ^ 
•NOTE: Use one page for each Evaluation Question. 

Page 1 of 2 
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Evaluation question (state): ' ' . J 1m4- A-t^W^^cA. 

^^^^^^^^ *** 




1. Was a clear answer obtained? (state) 

So Vo - f***< <+y* '^yjl- 

2. Unintended positive effects (if any): 



a. How valuable is this e£fect?%^ 




b. is the effect stable? 
3. Unintended negative effects (if any): 



• a. Hew serious is this effect? 



b. Is the problem resolvable in future applications? 



4. Overall nature of impact: w ^y^^^t ^T^f 
3o Ai4>U*til^, 

5. Conclusions (outline) : " - 



t Page 2 of 2 * 
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Finally, it was time for the team to draw conclusions about the 
dissemination service. They decided that the extent^ the dissemination 
resources was somewhat limited. Possible refinements would include the 
expansion of their Material Centers and the inclusion of other 
clearinghouses or ,inf ormation sources to supplement BRIC in their data 
searches! On the other ,hand, the utilization evidence was quite strong. 
The information being disseminated was making a difference on the local 
level. Because attention had been given to the planning, implementation 
and assessment stages of the impact evaluation, the team now had clear 
evidence supporting claims for the usefulness of their dissemination 
service. 
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SECTION FIVE ) 
SUMMARY 

A Disseminator's Guide for Evaluating Impact was written, primarily, 

with two groups of disseminators in mind. These are the State Capacity 

Building Project and Regional Exchange Project staffs. 

State Capacity Building Project clients include the host state 
agency, intermediate service agencies and local schools. Dissemination 
processes have included the' utilization of data bases, promising 
practices files, human resource files and the development of linkage ' ' 
systems. It is critical in evaluation planning to determine which 
processes, strategies and methods are being emphasized. 

Regional Exchanges are pafct of a system identified as the Research 
and Development Exchange. The major purpose is to aid school imprpvement 
by providing information, technical assistance and/or training; by 

"promoting the use of R&D outcomes? by promoting coordination among 
various groups; by increasing understanding and use of information about 
client heeds. Client groups include state, intermediate and local 
education agencies, as well as other labor atories. r Differences in client 
emphasis are important to note in developing^ impact evaluation plan. , 
Because it is not feasible to conduct impact evaluation on all 

- activities in which di^se^i#ators are involved, j£*c is ions have tp be made 
regarding, what ► to evaluate. Factors which are important to consider 
\nclude: 

f • Purpose of the evaluation 

• Recipients of the evaluation informition 
Accomplishment priority of objectives 

/ 
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• Likelihood of yielding useful information 

e Centrality of task or activity 

e Costa of conducting the evaluation 

Other considerations important to t£e development of an impact 
evaluation plan are the selection of appropriate units of analysis, 
determining whether the plan is reasonable and determining which efforts 
should be ongoing. 

A framework* was developed which is comprised of four major 
components. Included are the program description, evaluation focus, .data 
collection and analysis and impact assessment. 

The program description should .include four elements: 

e What is being disseminated * 

e To whom it is being disseminated 

e Purpose for^ which it "is being disseminated 



e Means by which it is being disseminated 

The evaluation focus should identify the audience for the evaluation 
information, delineate the questions to be addressed and define the data 
sources. In addition, the evaluation should provide a means for 
determining the degree tb which a prog 5^1 was implemented. 

Data collection tod analysis include^ a plan for jooll acting the data, 
but the plan must be* flexible enough to allow modification should such a 
need be created by unforeseen obstadles or problems. The data^ analysis 
involves cleaning trie data, analyaing the data as planned and reanalyzing 
the data in light of new relatdorfshipfe aaon§ U^^jjata which have become 
evident during the initial analysis.* 

Impact assessment is determined by answering four basic questions: 

e Were specific objectives achieved? 

e Were there any positive unintended effects? 
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• Were there any negative unintended effect*? 

• was the program implemented as intended? 
Conclusions from the assessment are derived from the questions: 

» How close did the actual impact come to the intended impact? 
e What factors facilitated achievement of objectives? Hindered? 

• What was the role of unintended effects? 

• What wad the overall success of the service? 

• Which segments should be repeated? Discontinued? Refined? 
The application of the framework is dependent upon the specific 

situation t existing A the program or projeq* to be evaluate^. That is, 
different dissemination programs emphasize different processes? the 1 
primary clients vary from pr^ram to program; who or which^agency is moi 
interested in the study varies across programs; each of these factors 
must be taken into account when actually developing an evaluation plan 
within the guidelines of the framework provided. 
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ANNOTATED LIST OP RESOURCES 



Abt Associates, Inc. "Rural America: A Social and Educational History of 
Ten Co*«unities" (sunnary) . June 1975. 

£ Deal, Terrence E..And Samuel C. Nutt. "Promoting, Guiding and Surviving 
Change in School Districts." Abt Associates, Inc., June- 1971. 

Herriott, Robert* E, "Federal Initiatives and Rural School Improvement: 
Findings From the Experimental Schools Program.", Abt Associates, Inc., 
Harch 1980. f 

The Abt Associates' longitudinal study of educational change in tural 
America was a five-year study of an experimental program in which ten 
small, rural school districts were funded (National Institute of 
Education) to improve the quality of their' local education. This 
extensive study of the Experimental Schools Program utilized %ight 
separate but coordinated efforts. The Abt Associates longitudinal study 
appears to exemplify an evaluation of change and impact. A study of such 
magnitude is not conducted easily or inexpensively; the wealth of 
* findings can make the effort Well worthwhile. 



Adams, Kay Angona and » Jerry P. Walker. Improving the Accountability of 
Career Education Programs: Evaluation Guidelines and Checklists* 
The Center for Vocational Education: Ohio State University, 1977.. 

This guide is one of a series of five handbooks intended to help local 
education personnel with measurement and evaluation. The handbook is 
v divided into thirteen stand-alone units and organized around an 
"Evaluatiorf' Prof ile." The profile is used to highlight critical 
ingredients for planning, implementing, communicating and using the 
results from evaluations of career education programs. The units 
themselves ax^e organized around checklists for assessing in detail the 
different aspects of an evaluation plan, instruments and reports. m 

Bank, Adrianne and Nancy C. Snidman. Guidebook for Evaluating 

Dlsseiti nation Activities: Resources for NDW Practitioners . - Center for 
the Study of Evaluation: UCLA, Los Angeles, 1981. 

. ; This guidebook was developed by the Center for the Study of Evaluation 
under a subcontract from THE NETWORK, Inc. The two-year effort involved 
participation of numerous persons involved with the National Diffusion 
Network, including numerous Developer /Demonstrators and State 
Facilitators. The guide appro£ihes evaluation as "a set of techniques 
for finding out what works" so that analysis of the data, however 
collected, can provide direction for improving the tasks of dissemination 
as defined By the NDN. „ 
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Bateman, Peter and Robert K. Yin. Targeting Educational Improvement 
Services (draft). The Case Study Institute. Washington, D.C. 
March 1981 

This report is one in a series of studies initiated in 1978 by 
Far West Laboratory for Educational Research and Development, The 
series expibres issues of information equity in education. This 
report examines one specific issue: whether educational Improvement 
services are being targeted to those in greatest need . The study is 
openly exploratory, designed to assess (1) whether future research 
would be fruitful and (2) how such research might be conducted. 
Seven mini-case studies of selected service improvement organizations 
(SIOs) , conduits by which publicly funded new 'ideas or materials are 
^distributed to local schools fo? implementation in the classroom, "are 
presented. The nummary section o^the report provides tentative 
lessons concerning four major questions: * 

• To what extent do SIOs engage in targeting to minority or needy 
users? 

• Is there a conflict for SIOs between the objectives of equity and 
improvement? 

• If targeting is done, what factors make it succeed? 

• If targeting is not done, what factors inhibit such a strategy? 

A set of recommendations about furthet research is drawn from the 
tentative lessons about these issues. Findings indicated that 
targeting does occur among the seven exemplary cases and that each 
§10 seemed in soae way tp be serving schools in -need. Two types of 
further research are reconmended: confirmatory studies and 
assistance studies (aimed at identifying ways of improving targeting 
among existing SlOs) ( . 

\ 

Faddis, Bonnie, Warren Evans, Maralyn Hartzell, Patricia Ruzicka and 
Kathryn Morimitzu. Handbook: Evaluating Program Implementation . 
BSEA Title I Evaluation technical Assistance Centers. Northwest 
Regional Educational Laboratory. Portland, Oregon. § 1981. 

The handbook was designed to assist school district personnel in 
designing and conducting an implementation study. A step-by-step 
approach to evaluating program implementation is presented. The 
handbook is divided into four sections, each designed to answer one 
of four planning questions as follows: 

1. Do you know what area of your program you are going to evaluate? 

2. Have you formulated specific evaluation questions? 

3. Do you know how you will collect information to answer your 
question (s) ? 



4. Do you know how you will^Ise information after it is collected? 

Extensive .appendices of data collection instruments* are included in 
the Handbook. 



Faddls, Bonnie and Marilyn Hartzell. Simulation Activities: Evaluating 
Program Implementation . ESEA Title I Evaluation Technical Assistance 
Centers.. Northwest Regional Educational Laboratory, Portland, 
Oregon. 1981. - * * * 

' This book accompanies the Evaluating Program Implementation Handbook 
and contains simulation activities which correspond to the 
explanations and examples presented in th^handbook. 

"Focused Review of Literature: Improving Educational Practice," 
Interorganlzatlonal Arrangements for Collaborative' Ef forte . 
Northwest Regional Educational Laboratory. .February 1980. 

This literature review, focused on defining and analyzing current 
findings related to thfe improvement of educational practice, found 
that achievement of improved practice was integrally related to the 
adoption of innovations abd to the dynamics of planned change. 

The literature review identified three factor* seen as critical to 
causing improved educational practice: 

• Appropriate information that addresses the real and perceived 
problems of practitioners 

• Acquisition. of new, personal skills necessary to carry out an 
implementation 

• Availability of skilled technical assistance in dealing with the 
political realities associated with a change effort 

" v J 

Hughes, Phillip, N. Russell, D. McConachy and W. Harden. Questions to Ask 
When Planning an Evaluation: A Procedure for Planning School 
Evaluation Where Access to Extensive Inservlcg or Extensive 
Consultant Assistance Is Not Possible . Teachers As Evaluators 
Project. Curriculum Development' Centre. Canberra, Australia, 
^March 1980. 

In the foreword, the authors state that "experience shows that when 
teachers actively participate in planning an evaluation, they develop 
a strong commitment to both the conduct of the evaluation and the 
implementation of its recommendations." This monograph was developed 
with that experience in mind. The monograph is divided into four 
sections: Resourc^Document 1 - "Questions to Ask When Planning an 
Evaluation," Resource Document 2 - "Curriculum Evaluation: an 
Introduction", Resource Document 3 - "Evaluation of Dimmesdale High 
School" and Section, 4 - "A Plan for Evaluating a Pastoral Care 
System. * 
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Klefert, James and Arthur Haser. Prfrjram/Project Evaluation, m p3e. 

tfniverslty of Washington Press"! Harch 1977 " ~ ™ * 

« 

This document is designed to be used as & workbook for planning the 
evaluation of a program or, project. As such, it can serve as a 
program planning guide as well as a program evaluating guide. 



Long, Susan A. and Randall A. Cognetta. Questionnaires; Their 
Development and Use , San Mateo County Office of Education, 
September 1978. ' 

* Questionnaires can be a useful tool for assessing the impact of 
dissemination activities. However, the construction of a 
questionnaire that will yield useful, valid results can be a 
difficult task. This monograph describes the construction and use of 
questionnaires. > 

Sections in the monograph include: (1) advantages and limitations of 
the questionnaire, (2) parts of p questionnaire, (3) pre-testing the 
tryout of a questionnaire, (4) the cover letter, (5) distribution, 
(6) getting them back and (7) compiling and analyzing responses. 



Louis, Karen Seashore, Sheila Rosentilum and James Molitar. Linking R&D 
With Schools — Strategies for Knowledge Use ^nd School Improvement: 
A Sunmary . Abt Associates, Inc., for the National .Institute of 
Education. July 1981. O 

This document presents an overview and major outcomes from a 
three-year &tudy of NIE's Research an\ Development Utilization (RDU) 
program. Legsbns, findings and implications from that program which 
may be*usefhl in designing future dissemination and school 
improvement activities are discussed. 



Madey, Doren L. k , et al . . Interim Report: An Evaluation of NIE's State 
pfoeeminatipn Grants Program . WTS Research Corporation. Aygust 1979 

The NTS Research Corporation's study is a multi-year effort sponsored 
by NIE's Prpgram on Dissemination and Improvement of Practice. The 
study of th£ capacity building component of the State Dissemination 
Grants Program is guided by two major questions: 

• Is dissemination capacity being built as a result of this 
Program? If so, how? * 

• Is the Program having an effect? If so, what is the nature of the 
* effect? m 

Of particular use to disseminators interested in evaluating impact of 
dissemination activities are the processes and procedures used by the 
NTS Research Corporation in this study. Information 4x\ process, 
procedures and instruments can be extrapolated through ana^sis of 
this and other reports resulting from the study. 
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pwen, John. The Impact of the Australian Science Education Project 

Schools . Curriculum Development Centre. Canberra, A-ustralia. 197& 

The Australian Sci.ence Education Project developed materials for 

• science education in grades 7-10 during 1969-1974. The project 

^ represented the first program of curriculum materials developed on a 

nationwide basis, and was fully supported by Australian and all State 
Departments of Education funds. 

This report is an account of the dissemination phase of ASEP in 

• 1975-1976. The study investigates the extent of the use of, the 

materials, the reasons for the first purchase of ASEP materials by 
schools; and^xamines barriers to the us* of t\\e materials. Factors , 
are analyzed both on a state-by-staJte basis and on the baSis of case 
studies of schools in each state. 

• Because the study was concerned with a range of questions related to 

the impact of ASEP materials on Australian schools, a combination of 
methods was used to obtain data. Information was collected from 
government department records, from historical documents, and through 
interviews with teachers and other persons who were potentially 
influential in the diffusion of ASEP materials. 

Roberts, Jane M. E. Implementation of Innovations in Educational 

Organization and Instruction; Working Paper . Research for Better 
Schools, Inc. June 1978 

• This paper describes numerous models, studies and research findings 

related to* the Implementation of innovations in educational 
organizaton and instruction. The implications of research on the 
processes, influences and effects considered to be important 
variables in the implementation of educational innovations are 
discussed. ' 

• 

Schmidt, Richard E., John W. Scanlon and James B. Bell. Evaluabillty 
Assessment: Making Public Programs Work Better . Project SHARE. 
Human Services Monograph Series, No. 14. November 1979. 

• This monograph was designed to describe techniques that can be used 

in making public programs work more effectively. The document is 
targeted toward evaluators who, are charged with informing management 
and policy officials about the effectiveness of their programs- The 
method described is intended to make pgfcible a closer link between 
management and evaluators, in the hope that the net result will be 

• better program management and more effective programs. 

-Evaluabillty Assessment* (EA) is a descriptive and analytic process 
intended to produce a reasoned basis for proceeding with an evalua- 
tion of use to both management and policymakers. The process was 
developed by the members of the program evaluation group of the Urb<yf 

• Institute between 1960 and 1978. The object of an EA is to arrive at 

a program description that can be evaluated— a description that 
permits 4 program to be measured with some reasonab^^ssurance that 
the evaluation can be done and that predetermined ex^tations can be 
realized. 
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Sieber,"Sam D. Knowledge for What?: An Eva l inr ^MP" 1 1 1 

DisaW nation System , U.S*. Virgin Island's, uepSIRnent of Education. 0 
January 1981. • • ( . 

This repoftujpresents the pr oced ures used in and results obtained from 
a survey-of primary clients' satisfaction with the Virgin Islands * m 
Educational Dissemination System, Department 6f Education duj^Lng its 
first year of operation. The Vitgin Islands dissemination project is 
a State Capacity Project which uses full-time "linkers" (field 
agents) who meet w^th clients at all levels of the educational — 
^ system, learn about *the information needs, refer the client's request 
to a^computerized retrieval agency in thtstates, and deliver the 
information to clients when it arrives. special purpose*of the 
*ttfdy w^s to determine whether the Impact of the service could be 
enhanced by concentrating on certain promising toi^ent^* j[a procedure 
known as market segmentation) . Xhe report r^^en^jta a System 
Assessment Survey designed by -the author to* be used Tq,the evaluation. 
§t State Capacity Ijuilding Projects. The Survey Js au^ilable from 
^he author upon request." » 



Smifcfi, Nick L., editor. Metaphors for Evaluation: Sources of New 
Methods . .Volume t, New Perspectives in- Evaluation; 
f Sa^e Publi&tions, Inc. Beverly Hills, California. 1981. 
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This bo^k reports on the results of seven efforts to identify new 
methods for use' in evaluation by studying existing procedures used in 
other disciplines. 



~ Nev^Teghniques for Evaluation . Volume II, New Perspectives 
iri Evaluation. Sage Publications, Inc. Beverly Hills, California.^ 
1981. * 

This volume introduces a range -of new evaluation techniques &d 
discusses their application. Included are cost analysis methods, 
-^exploratory data analysis methods, criticism methods for appraising^ 
ewjftuation studies, product evaluat^.* methods and journalistic* ^ 
methods. 



Weber.* Margaret B. "An Evaluatibn Model for Innovative Individualized 
Programs," Studies in Educational Evaluation , Vol. 3, No. 2. Summer 
* 1977. (Pp. 87-93.) " % 

This eyrticle describes the tri-level pr<3fcess model used in the 
evaluation of an innovative secondary program. Findings and implica- 
tions fofr the evaluation of innovative programs are discussed. It 
was found that Level I (learner diagnosis^ and development) and Level 
III (evaluation of the program as compared against a criterion 
program) of the program evaluation process did not yield usabLe 
data. Level II (where the innovative; program i»-^}u£ged against its 
own objectives) was\the only area in which traditional evaluation 
/ methods were productive. - . 
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Weter suggests that the issue of "eff ectiverAs" must be determined 
internal to the program to be evaLuated if commitment to responsive 
change is indeed important. She points out that it is this thrust 
that emphasizes "needs assessment" 'in evaluation, as needs assessment 
is a synonym for the identification of expressed drives. Therefore, 
program ef f ectivegfbss can be determined only by assessing ^.the "fit" 
between program outcora%iand expressed needs. 




